Podman support and option to backup across all projects #66

AlexMcDermott · 2025-09-16T07:34:15Z

Hi, cool project

[alexmcdermott@grom:~/homelab/containers/stack-back]$ podman exec -it stack-back rcb backup
Traceback (most recent call last):
File "/restic-compose-backup/.venv/bin/rcb", line 10, in <module>
sys.exit(main())
^^^^^^
File "/restic-compose-backup/restic_compose_backup/cli.py", line 23, in main
containers = RunningContainers()
^^^^^^^^^^^^^^^^^^^
File "/restic-compose-backup/restic_compose_backup/containers.py", line 411, in **init**
if container_data.get("Id").startswith(os.environ["HOSTNAME"]):
~~~~~~~~~~^^^^^^^^^^^^
File "<frozen os>", line 714, in **getitem**
KeyError: 'HOSTNAME'

Podman network issue

This was triggered when the internal backup container is created. Because it was trying to use the main backup containers network, without a pod being specified, and not started as part of the same compose file.

[alexmcdermott@grom:~/homelab]$ podman exec -it stack-back rcb backup
2025-09-15 21:55:10,130 - INFO: Starting backup container
2025-09-15 21:55:11,391 - ERROR: 500 Server Error for http+docker://localhost/v1.41/containers/create: Internal Server Error ("container create: container dependency b68b86daed653b6a34f74873cf14ca0550bda075cb7c21d4a5354f82ee0b84a9 is part of a pod, but container is not: invalid argument")
Traceback (most recent call last):
File "/restic-compose-backup/.venv/lib/python3.12/site-packages/docker/api/client.py", line 275, in *raise*for_status
response.raise_for_status()
File "/restic-compose-backup/.venv/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/containers/create

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/restic-compose-backup/restic_compose_backup/cli.py", line 167, in backup
result = backup_runner.run(
^^^^^^^^^^^^^^^^^^
File "/restic-compose-backup/restic_compose_backup/backup_runner.py", line 20, in run
container = client.containers.run(
^^^^^^^^^^^^^^^^^^^^^^
File "/restic-compose-backup/.venv/lib/python3.12/site-packages/docker/models/containers.py", line 876, in run
container = self.create(image=image, command=command,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/restic-compose-backup/.venv/lib/python3.12/site-packages/docker/models/containers.py", line 935, in create
resp = self.client.api.create_container(**create_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/restic-compose-backup/.venv/lib/python3.12/site-packages/docker/api/container.py", line 440, in create_container
return self.create_container_from_config(config, name, platform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/restic-compose-backup/.venv/lib/python3.12/site-packages/docker/api/container.py", line 457, in create_container_from_config
return self._result(res, True)
^^^^^^^^^^^^^^^^^^^^^^^
File "/restic-compose-backup/.venv/lib/python3.12/site-packages/docker/api/client.py", line 281, in *result
self.*raise_for_status(response)
File "/restic-compose-backup/.venv/lib/python3.12/site-packages/docker/api/client.py", line 277, in *raise*for_status
raise create_api_error_from_http_exception(e) from e
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/restic-compose-backup/.venv/lib/python3.12/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.41/containers/create: Internal Server Error ("container create: container dependency b68b86daed653b6a34f74873cf14ca0550bda075cb7c21d4a5354f82ee0b84a9 is part of a pod, but container is not: invalid argument")
2025-09-15 21:55:11,396 - INFO: No alerts configured

lawndoc

Nice simplification of the container filter loop. Please see my comment about the shared network stack

src/restic_compose_backup/backup_runner.py

lawndoc · 2025-09-24T04:18:29Z

more reliable to use the hostname from the socket library instead.

This is fine, using socket should be backward compatible

lawndoc · 2025-10-18T01:33:10Z

Doing some testing of this PR now, sorry it's taken me so long to get to this. Everything is going well in my previous test setup. Looks like the container filter changes will also fix #62 which is a nice bonus.

Still need to test:

database backup with multiple docker networks after the changes to the backup container's network stack
the new config variable to backup all projects with one backup container

lawndoc · 2025-10-18T01:46:32Z

@AlexMcDermott your code is working as expected for individual projects, but it fails when backing up databases in other compose projects. This is because docker compose will create different networks for each compose project.

You can import other networks into your compose project, but that's not a good user experience.

Instead, I think we can give the backup container access to every network that it will need access to, instead of just the network defined for the main backup container. I actually like this better anyways, because then we don't have to mess with the backup container's network even within a single compose project.

src/restic_compose_backup/containers.py

src/restic_compose_backup/backup_runner.py

src/restic_compose_backup/cli.py

src/restic_compose_backup/containers.py

src/restic_compose_backup/backup_runner.py

AlexMcDermott · 2025-11-01T00:33:33Z

Hi @lawndoc thanks for the review. I've implemented your feedback and tested it out with a Postgres container started in another compose stack. Let me know what you think : )

lawndoc · 2025-11-04T04:37:24Z

Great work, looks like the right approach now.

Looks like tests are failing. I don't have time to look closely right now but at a glance it looks like the mocks maybe aren't generating a valid network config for the __init__ method. If so, the easiest fix is to just add if networks else {} to the end of the self._network_details declaration

AlexMcDermott · 2025-11-04T05:09:01Z

Cheers, thanks for checking it out. Now I know its looking decent I'll fix up the tests

AlexMcDermott · 2025-11-08T23:52:54Z

Cheers, thanks for checking it out. Now I know its looking decent I'll fix up the tests

Should be working now, added network settings to the fixture

lawndoc

Looks good, thanks for the contribution!

iTrooz · 2025-11-16T17:33:15Z

I believe there was a regression

Here are my tests, making a backup using podman:
v1.5.0 (when this was merged):

  File "/restic-compose-backup/.venv/lib/python3.12/site-packages/requests/adapters.py", line 682, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

v1.5.1: working
v1.5.2: working
v1.5.3: working
v1.5.4:

  File "/restic-compose-backup/.venv/lib/python3.14/site-packages/docker/errors.py", line 39, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation) from e
docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.41/containers/create: Internal Server Error ("container create: container dependency 0a450c8395cf8c961da567cb65eb6d127225b125b1d39607e835cffdc2601d57 is part of a pod, but container is not: invalid argument")

lawndoc · 2025-11-17T16:09:09Z

You're right, I totally forgot what the original issue was for this PR when I fixed socket proxy issue in 1.5.4

I'll create a new issue and fix the regression with the approach from this PR.

I'll look into adding an integration test for podman as well.

AlexMcDermott added 6 commits September 15, 2025 22:08

remove unused logs

8e07295

gracefully handle signals

5086395

created backup container just uses own stack

f29307a

switch to use socket lib to get hostname

c95caf6

Add back ability to run over all projects

d6f00d3

add back label

6ce96af

lawndoc reviewed Sep 24, 2025

View reviewed changes

src/restic_compose_backup/backup_runner.py Show resolved Hide resolved

lawndoc mentioned this pull request Sep 24, 2025

Multiple Compose Projects: RuntimeError: Backup process already running #62

Closed

AlexMcDermott added 7 commits September 27, 2025 06:56

update comment to match command

5823127

remove unused import

f59f7b5

ruff

f53acc6

add hostname mocking

2da8dca

ruff

c504b3e

switch to use same network not same interface

164fda0

switch to use ip for db connection

36ec179

AlexMcDermott requested a review from lawndoc September 27, 2025 10:11

AlexMcDermott marked this pull request as ready for review September 27, 2025 10:11

AlexMcDermott changed the title ~~Podman support and option of backup across all projects~~ Podman support and option to backup across all projects Oct 15, 2025

lawndoc requested changes Oct 18, 2025

View reviewed changes

src/restic_compose_backup/containers.py Outdated Show resolved Hide resolved

src/restic_compose_backup/containers.py Outdated Show resolved Hide resolved

src/restic_compose_backup/containers.py Outdated Show resolved Hide resolved